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FOREWORD 

This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  rending  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  in  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 


Preceding  page  blank 
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T&E  GUIDELINES  FOR  AIRBORNE  GENERAL  SURVEILLANCE  RADAR  SYSTEMS 


The  checklist  items  presented  here  are  specifically  applicable  to  air¬ 
borne  general  surveillance  radar  testing  and  evaluation.  It  is  suggested 
that  the  user  of  this  volume  also  refer  to  the  Report  of  the  Defense  Science 
Board  on  Test  and  Evaluation  which  contains  general  checklist  items  also 
applicable  to  this  system  T&E  program.  The  checklist  items  presented  here 
are  organized  into  time  phases  of  the  acquisition  process  oriented  to  the 
DSARC  cycle. 

The  checklists  cover  various  aspects  of  the  major  activities  that 
should  be  underway  during  a  given  time  period.  Hence,  a  checklist  might 
cover  the  (1)  evaluation  of  work  that  occurred  in  the  previous  phase, 

(2)  conduct  of  tests  planned  in  the  previous  phase  and  executed  in  the 
subject  phase,  and  (3)  plans  and  other  preparatory  actions  for  test  sche¬ 
dules  to  be  conducted  in  a  subsequent  phase.  For  reasons  such  as  this, 
items  on  some  subjects,  such  as  development  test  plans,  may  appear  in  more 
than  one  phase.  In  addition,  since  the  Services  and  the  DSARC  have  flexi¬ 
bility  in  deciding  how  rapidly  to  progress  in  the  validation  phase,  there 
may  be  cases  where  the  Request  for  Proposals  (RFPs) ,  proposal  evaluations, 
source  selections,  or  contract  negotiations  may  occur  after  the  DSARC 
approves  full-scale  development  instead  of  before.  For  this  reason,  it 
is  recommended  that  previous  checklists  in  the  Validation  Phase  be  re¬ 
viewed  when  entering  the  Full-Scale  Engineering  Development  Phase.  The 
following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceptual  Phase  of  the  acquisition  of  the  system. 

This  phase  (often  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  development  of  a  weapon  system  con¬ 
cept  that  offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  for  in  DoD  Directive  5000.1  specifically,  the 
objectives  of  this  phase  should  be: 
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1.  To  verify  that  there  is  a  military  need  for  the  proposed 
system. 

2.  To  demonstrate  that  there  is  a  sound  physical  basis  for  a  new 
weapon  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need. 

4.  To  show  that  the  proposed  solution  is  superior  to  its  com¬ 
petitors  in  terms  of  potential  effectiveness,  probability  of 
success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  need  to 
show  that  it  is  better  to  start  advanced  development  now 
rather  than  to  wait  for  future  technological  improvements. 

6.  To  identify  the  key  risk  areas  and  critical  issues  that  need 
to  be  resolved  before  full-scale  development  is  initiated. 

The  most  important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  tine  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSARC  recommends  full- 
scale  development  of  the  system) . 

While  these  objectives  are  not  spelled  out  in  the  DoD  Firective  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm: 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of 
development  costs,  preliminary  estimates  of  life  cycle  costs 
and  potential  benefits  in  context  with  overall  DoD  strategy  and 
fiscal  guidance. 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  and  solutions  are 
in  hand. 

4.  Realism  of  the  plan  for  full-scale  development. 


2 


In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&E  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the 
T&E  interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must 
include  adequate  guidance  to  ensure  that  sufficient  resources  and  time  are 
available  so  that  engineering  effort  can  properly  support  the  initial  DT&E 
with  hardware,  software,  technical  data,  and  Lraining. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  4  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  incorporated  in  the  engineering  development  contract  as  well  as  the 
DT&E  associated  with  addressing  the  critical  issues  and  areas  of  major 
risk  identified  in  the  DCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


The  checklist  items  contained  in  this  phase  are  for  guidance  in 
conducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  pro¬ 
duction  decision.  By  this  time,  the  system  is  well-defined  and  is 
becoming  a  unique  item  and,  hence,  sound  judgment  must  be  applied  in  using 
these  checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have: 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development 
process  is  complete  and  that  the  design  risks  have  been  mini¬ 
mized  (the  system  is  ready  for  production). 

2.  Demonstrate  that  the  system  will  meet  specifications. 
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The  primary  objectives  of  the  IOT&E  should  be  to: 

3.  Assess  operational  suitability  and  effectiveness. 

A.  Validate  organizational  and  employment  concepts. 

5.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  production/deployment 
will  be  recommended  to  the  Secretary  of  Defense. 

The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The 
IOT&E  plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed 
to  evaluate  the  inputs  associated  with  the  primary  objectives.  These 
tests  should  not  be  conducted  until  the  primary  objectives  cf  the  DT&E 
have  been  met.  Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E 
major  milestone — the  system  is  ready  for  production — has  been  achieved. 
Close  monitoring  of  the  T&E  Service  activity  is  required  during  the  latter 
stages  of  this  phase. 

SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the  early 
production  items. 

To  enter  the  Production/Deployment  Phase,  the  DSARC  will  have  re¬ 
viewed  the  program  to  confirm: 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of 
production  and  logistic  problems  is  complete. 

•  All  technical  uncertainties  have  been  resolved  and  opera¬ 
tional  suitability  has  been  determined  by  T&E. 

•  The  realism  of  the  plan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 
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2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the 
previous  phase;  however,  much  of  the  testing  is  verification  that  the 
production  system  performance  is  as  expected.  Hence,  most  of  the  items 
in  the  previous  phase  are  appropriate  to  this  phase,  especially  those 
related  to  OT&E. 


j  I •  CONCEPTUAL  PHASE 

During  this  phase  the  program  is  being  conceived  and  the  DCP  or  its 

equivalent  is  being  prepared.  The  test  and  evaluation  checklist  covers 
the  following: 

1.  Test  Program/Total  Costs 

2.  Test  Facilities  and  Instrumentation 

3.  Operational  Scenario 

4.  Combining  Untried  Systems 

5 .  Documentation 


Preceding  page  blank 
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1. 


TEST  PROGRAM/TOTAL  COSTS 


Prior  to  DSARC  I,  all  the  phases  of  the  test  program  should  be  ad¬ 
dressed  so  that  the  approximate  total  costs  and  development  schedules 
include  consideration  of  all  likely  activities  In  the  over-all  pro¬ 
gram. 

In  an  airborne  general  surveillance  radar  program,  these  might  include 
the  cost  for  range  support  during  the  R&D  effort,  new  test  range  instru¬ 
mentation  or  facility  requirements,  the  cost  of  the  OT&E  program  (both 
IOT&E  and  follow-on  OT&E  including  test  gap  closing  provisions) ,  and  the 
cost  of  special  tests,  such  as  special  antenna  ranges  for  measuring  ultra- 
low  antenna  side  lobes,  scale  modelling  to  obtain  preliminary  data  on 
interaction  between  the  antenna  and  aircraft,  and  special  simulators  to 
exercise  the  ECCM  features  of  the  radar  system. 

2.  TEST  FACILITIES  AND  INSTRUMENTATION 

Before  DSARC  I, test  facilities  and  instrumentation  requirements  to 
conduct  operational  tests  should  be  identified,  along  with  a  ten¬ 
tative  schedule  of  test  activities. 

The  applicability  of  the  test  ranges,  the  adequacy  of  the  facilities 
and  instrumentation,  and  the  availability  (at  the  appropriate  time)  of 
real  or  simulated  enemy  jammers  should  be  verified.  Insofar  as  possible 
alternative  approaches  (different  ranges,  etc.)  and  necessary  instru¬ 
mentation  improvements  should  be  specified.  Of  prime  importance  are  the 
constraints  to  be  placed  on  the  test  because  of  range,  instrumentation, 
and  target  radar  availability.  If  these  factors  are  found  to  cast  sig¬ 
nificant  doubt  on  the  meaningfulness  of  the  test  data  because  of  a  lack 
of  operational  realism,  the  steps  necessary  to  assure  meaningful  data 
should  be  identified  and  planned  before  inclusion  in  the  DCP. 

3.  OPERATIONAL  SCENARIO 

Analytic  and  empirical  studies  should  be  conducted  prior  to  DSARC  1 
to  insure  that  the  range  of  critical  performance  characteristics 
has  been  specified . 
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Each  performance  characteristic,  such  as  clutter  rejection  require¬ 
ments,  vulnerability  to  countermeasure,  mutual  interference,  etc., 
specified  should  be  measurable  through  laboratory  or  full-scale  testing. 

The  test  plan  and  the  number  of  tests  should  b%  prepared  so  that  meaning¬ 
ful  results  and  conclusions  can  be  drawn  (at  suitable  confidence  limits 
if  necessary).  If  a  given  target  detection  range  is  specified,  testing 
at  one  or  two  closing  and  opening  velocities,  at  two  or  three  altitudes, 
in  two  or  three  different  clutter  environments  may  satisfy  the  requirements 
for  meaningful  test  results.  Testing  in  advanced  development  should 
be  planned  to  explore  the  radar  detection  and  clutter  rejection  perfor¬ 
mance  over  a  broad  range  of  terrains  and  altitudes  so  as  to  provide  insight 
into  system  performance  over  the  expected  operational  range  and  not  just  at 
a  single  point.  Inputs  from  the  ORT  (Overland  Radar  Technology)  program 
should  be  used  to  provide  clutter  data. 

4.  COMBINING  UNTRIED  SYSTEMS 

Whenever  possible,  try  to  avoid  combining  untried  systems  prior  to  the 
initiation  of  a  radar  test  program. 

£ 

A  radar  system  developed  for  use  in  a  fighter  aircraft  incorporated 
as  a  new  feature  a  built  in  test  equipment  (BITE)  system.  This  permitted 
onboard  checking  and  troubleshooting  of  the  radar.  Since  the  system  was 
*iew,  it  was  not  too  well  understood  initially  and  many  problems  arose  in 
connection  with  the  BITE.  Much  development  time  and  effort  required  to 
get  the  BITE  working  properly  took  up  time  that  should  have  been  spent  on 
the  development  of  the  radar  itself. 

5.  DOCUMENTATION 

Prepare  adequate  documentation  to  provide  full  benefit  from  development 
history  for  future  programs.  Use  this  documentation  in  developing  T&E 
plans  for  subsequent  programs. 

In  the  development  of  one  new  l idar  system  (which  extended  over  almost 
a  15-year  period),  no  effort  had  ever  been  made  to  orovide  overall  documen¬ 
tation  of  the  program.  Historical  development  of  the  system  had  never  been 
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thoroughly  documented  and  no  attempt  was  made  to  conduct  a  scientific 
analysis  of  the  data.  The  only  extensive  analysis  of  the  data  was 
limited  to  systems  performance  evaluation  prior  to  the  determination  of 
the  procurement  source.  As  a  result  the  lessons  learned  on  this  program 
were  not  adequately  documented  and  hence  not  available  for  the  development 
of  future  similar  systems  testing  and  evaluation  plans. 
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II.  VALIDATION  PHASE 

% 

In  the  Validation  Phase  the  issues  raised  by  the  DCP  may  be  resolved 
by  conducting  tests  on  the  system.  The  development  and  operational  plans 
will  be  defined  in  considerable  detail.  Additional  test  and  evaluation 
checklist  items  are: 

1.  Compatibility  of  Components 

2.  Software 

3.  Flight  Testing 

4.  Testing  in  the  Presence  of  Clutter 

5.  Brassboard  Stage  Changes 

6.  Antenna  Patterns 

7.  Simulations 

8.  Testing  Objectives 

9.  Reliability  Testing  Plan 

10.  Test  Scheduling 

11.  Progressive  Testing 

12.  Demonstration  Testing 

> 

The  reader  should  review  the  previous  checklist  items  since  many  of 
them  will  be  appropriate  to  this  phase. 


1.  COMPATIBILITY  OF  COMPONENTS 


Be  sure  that  items  which  must  work  compatibly  In  the  airborne  general 
surveillance  radar  system  are  developed  as  a  unit.  This  is  particularly 
important  for  radar  transmitters  and  transmitter  tubes. 

One  manufacturer  used  some  long  lead  time  money  to  develop  a  power 
transmitter  in  cooperation  with  another  manufacturer  who  was  concurrently 
developing  the  needed  klystron  tube.  This  permitted  testing  and  demon¬ 
stration  of  the  high  power  transmitter  at  an  early  date  in  an  acceptable 
configuration.  In  a  number  of  other  cases  the  transmitter  tube,  which  has 
frequently  been  the  pacing  element,  was  funded  independently  before  the 
procurement  of  the  radar.  Interface  problems  and  a  loss  of  flexibility  in 
trade-offs  between  the  tube  and  the  transmitter  almost  invariably  resulted. 

2 .  SOFTWARE 

Test  and  evaluation  should  ensure  that  software  products  associated 
with  the  ECM  and  data  processing  subsystems  are  tested  appropriately 
during  each  phase. 

Software  has  often  been  developed  more  as  an  add-on  than  as  an  integral 
part  of  the  overall  system.  Software  requirements  need  the  same  considera¬ 
tion  as  hardware  requirements  in  the  Validation  Phase.  Usual  practices 
often  do  not  sufficiently  provide  for  testing  the  software  subsystem  con¬ 
cept.  Often  the  facilities  available  to  contractors  for  software  develop¬ 
ment  and  verification  are  critical  to  schedule  and  cost.  Failure  to 
thoroughly  check  out  the  software  on  several  ECM  programs  resulted  in  in¬ 
ability  to  properly  classify  certain  emitters. 


3.  FLIGHT  TESTING 

Engineering  flight  tests  should  be  conducted  on  the  entire  radar  sys¬ 
tem  as  soon  as  possible. 
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An  airborne  general  surveillance  radar  must  be  tested  in  its  natural 
airborne  environment.  Each  new  installation  and  configuration  is  unique 
and  must  be  reevaluated.  A  system  cannot  be  tested  on  the  ground  and 
shipped  for  field  installation;  flight  testing  is  also  necessary.  Test 
bed  aircraft  should  be  used  for  early  flight  testing.  In  these  tests, 
the  transmitter  mounting  configuration  and  antenna  pointing  directions 
relative  to  the  aircraft  should  be  carefully  selected  to  ensure  that 
realistic  clutter  rejection  tests  are  performed.  The  system  must  have 
excellent  clutter  rejection  performance  for  the  automatic  detection  system 
to  operate  properly.  Flight  tests  should  be  flown  over  representative 
terrain  so  that  meaningful  conclusions  can  be  drawn  from  the  data.  During 
these  tests,  it  is  important  to  provide  instrumentation  to  record  the 
critical  functions;  reliance  on  the  radar  operator's  memory  of  what 
happened  should  be  avoided. 

4.  TESTING  IN  THE  PRESENCE  OF  CLUTTER 

Testing  of  an  airborne  general  surveillance  radar  system  should  occur 
in  the  presence  of  actual  clutter.  This  means  that  an  airborne  test 
is  mandatory  using  typical  or  representative  terrain  to  provide  real 
clutter  information. 

If  the  parameters  which  affect  clutter  (primarily  frequency,  pulse- 
width,  and  PRF)  are  essentially  unchanged  from  a  proven  design,  airborne 
testing  requirements  are  minimal.  If  any  of  these  parameters  is  changed 
significantly,  a  major  test  program  should  be  initiated  at  the  earliest 
possible  date. 

The  use  of  a  synthetic  target  generator  is  strongly  recommended  in 
conjunction  with  the  evaluation  of  actual  clutter  effects.  It  is  neces¬ 
sary  to  use  at  least  one  live  target  as  a  reference.  Local  aircraft 
traffic  which  may  easily  number  into  the  hundreds  is  also  valuable  in 
evaluating  system  performance.  Additional  synthetic  targets  provide  in¬ 
formation  on  the  behavior  of  the  system  over  the  full  performance  envelope 
of  the  system  without  tne  need  for  extensive  external  preparation  and 
extensive  time  requirements.  Note  that  the  aircraft  continues  to  be  flown 
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in  typical  flight  profiles  so  that  the  actual  ground  clutter  will  be  pre¬ 
sent  in  evaluating  the  synthetic  targets. 

5.  BRASSBOARD  STAGE  CHANGES 

During  the  brassboard  stage  of  a  radar  development  program  provision 
should  be  made  for  changes  in  software  which  can  be  performed  easily 
and  in  timely  fashion. 

As  an  example  of  what  can  be  done,  one  manufacturer,  during  a  compet¬ 
itive  fly-off  which  lasted  nine  months,  made  a  parametric  change  in  their 
radar  on  the  average  every  day  and  a  major  parametric  change  on  the  average 
of  every  second  day.  A  major  contributor  to  this  performance  was  a  design 
using  digital  intercommunication  between  chassis,  and  a  bus  architecture 
similar  to  that  now  used  in  many  computers.  In  addition  to  allowing  fast 
change  of  radar  parameters  with  a  software  change,  it  also  avoided  major 
rewiring  of  the  aircraft  (a  problem  for  relatively  minor  modifications 
throughout  the  operational  life  of  a  radar).  Another  consideration  in 
this  performance  was  a  design  which  had  been  proven  reliable  so  that  no 
significant  time  was  lost  on  repairs.  This,  in  turn,  was  due  to  use  of 
highly  reliable  components  and  provisions  for  in-aircraft  servicing. 

6.  ANTENNA  PATTERNS 

Measure  antenna  patterns  in  all  situations  which  are  meaningful  for 
operation. 

Antenna  patterns  should  be  measured  on  an  antenna  range,  on  the  air¬ 
craft  in  which  the  radar  is  to  be  flown,  both  with  and  without  IFF  opera¬ 
tional.  Antenna  testing  should  also  include  evaluation  of  cooling  and  beam 
stabilization  if  used. 

7.  SIMULATION 

Analysis  and  simulation  should  be  conducted,  where  practicable,  before 
each  phase  of  development  flight  testing. 
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Analysis,  simulation  and  other  ground  testing  should  be  used  to  predict 
test  outcome  and  to  establish  test  objectives.  The  flight  test  may  then  be 
accomplished  to  achieve  the  objectives.  Comparison  of  simulation  and  flight 
test  results  provides  better  understanding  of  the  system.  One  radar  manu¬ 
facturer  first  developed  a  simulator  for  their  radar,  then  combined  it  with 
a  war  game  model  to  evaluate  the  radar  system.  This  approach  was  of  great 
use  to  the  manufacturer  in  evaluating  the  impact  of  proposed  changes  in  the 
radar  relative  to  the  overall  effectiveness  of  the  system. 

8.  TESTING  OBJECTIVES 

Test  conditions  during  validation  testing  should  be  determined  by  the 
primary  objectives  of  that  test,  rather  than  by  more  general  considera¬ 
tions  of  realism,  etc. 

Whenever,  in  the  interest  of  obtaining  advanced  engineering  data,  a 
non-tactical,  non-operational  configuration  is  required  for  testing,  the 
results  of  the  tests  should  not  be  challenged  by  the  fact  that  the  configu¬ 
ration  was  not  tactical  or  operational.  For  example,  if,  in  the  develop¬ 
ment  of  an  airborne  general  surveillance  radar  system,  problems  with  the 
signal  processor  preclude  adequate  clutter  rejection,  it  may  be  desirable 
to  use  a  target  equipped  with  a  transponder  as  an  aid  to  testing  the  other 
aspects  of  the  radar  system. 

9.  RELIABILITY  TESTING  PLAN 

A  reliability  testing  plan  should  be  included  as  a  part  of  the  T&E 
plan  for  an  airborne  general  surveillance  radar. 

In  order  to  provide  reasonable  confidence  in  the  reliability  of  an  air¬ 
borne  general  surveillance  radar  system,  a  large  test  sample  size  or  a  long 
test  period  is  indicated.  A  study  of  the  tradeoffs  among  sample  size,  test 
duration,  and  the  confidence  level  should  accompany  any  reliability  testing 
plan.  If,  for  some  reason,  the  sample  size  or  test  duration  is  less  than 
desirable  (which  is  the  normal  situation),  a  plan  for  showing  how  the  relia¬ 
bility  goals  are  to  be  demonstrated  must  be  prepared. 
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10.  TEST  SCHEDULING 


Before  DSARC  II,  the  nature  of  the  schedule  for  the  OT&E  plan  should 
be  addressed. 

In  general,  the  OT&E  plan  should  include  engagements  in  the  environ¬ 
ments  in  which  the  new  system  is  expected  to  operate.  Airborne  general  sur¬ 
veillance  radar  testing  may  be  addressed  in  several  phases,  such  as: 

(a)  One-on-one  testing  against  available  simulators  of  the 
assumed  targets. 

(b)  Single  aircraft  detection  and  tracking  in  a  jamming  environment. 

(c)  Multiple  aircraft  detection  and  tracking  in  an  interference 
environment  caused  by  other  radars  or  electronic  equipment 
as  well  as  jammers. 

(d)  Comparative  testing  of  the  new  radar  system  with  existing 
similar  systems  to  demonstrate  the  increased  capability. 

Test  range  and  resource  requirements  should  be  estimated,  and,  if  inter¬ 
service  testing  is  contemplated,  preliminary  plans  for  such  testing  should 
be  coordinated  with  the  cooperating  service. 

11.  PROGRESSIVE  TESTING 

The  design  of  the  set  of  tests  to  demonstrate  system  feasibility 
prior  to  DSARC  II  should  be  based  on  a  building  block  concept. 

High  technical  risk  items  should  be  tested  early  with  subsequent  tests 
incorporating  more  of  the  hardware  until  the  complete  system  concept  has 
been  demonstrated  feasible.  For  example,  if  the  high  risk  item  is  a  high 
power  tube,  then  the  demonstration  of  tube  performance  should  be  conducted 
prior  to  the  transmitter  system  test. 

12.  DEMONSTRATION  TESTING 

Major  tests  must  be  accomplished  and  system  feasibility  adequately 
demonstrated  before  the  system  is  allowed  to  move  to  the  next  phase 
of  the  acquisition  process. 

For  example,  before  the  system  is  allowed  to  go  into  the  IOT&E  phase, 
all  engineering  tests  should  be  complete;  before  the  Full-Scale  production 
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phase  is  entered,  the  IOT&E  of  the  total  airborne  general  surveillance 
radar  system  should  have  been  successfully  performed  by  the  using  com¬ 
mand;  prior  to  deployment  of  the  system  to  the  user,  successful  com¬ 
pletion  of  the  acceptance  testing  of  the  initial  production  items  must 
have  occurred. 
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III.  FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


In  this  phase,  the  T&E  plans  developed  in  the  Validation  Phase  will 
be  refined  and  the  development  testing  conducted.  IOT&E  plans  will  simi¬ 
larly  be  refined;  personnel  will  be  assigned  and  trained  and  the  IOT&E 
conducted.  The  full-scale  development  checklist  includes: 

1.  Radar  System  Performance  Evaluation 

2.  Avoiding  Testing  Delays 

3.  Temperature  Cycle  Tests 

4.  Interfaces  with  Other  Systems 

5.  Mechanical  Components 

6.  Ground  Test 

7.  Maintenance  of  Flexibility 

8.  Value  of  Envelope  Expansion 

9.  Maintenance  Data  Package 

10.  Production  Decision 

It  is  suggested  that  the  checklists  for  the  previous  phases,  particu¬ 
larly  the  Validation  Phase,  be  reviewed  by  those  interested  in  a  checklist 
for  full-scale  development. 
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1.  AIRBORNE  GENERAL  SURVEILLANCE  RADAR  SYSTEM  PERFORMANCE  EVALUATION 

Evaluate  the  performance  of  the  radar  system  Including  signal  pro¬ 
cessing  capabilities  and  Interaction  of  the  high  power  tube  and  the 
transmitter,  taking  into  account  all  related  factors. 

This  includes  tracking,  taking  into  account  main  beam  clutter,  digital 
to  analog  conversion,  subclutter  visibility  including  coherency  (transmitter 
stability),  amplitude  stability  and  filter  sidelobes,  sidebands,  bandwidth 
(both  instantaneous  and  operating),  frequency  agility,  PRF  agility,  pulse 
compression,  software  (both  programmable  and  hardware),  range  velocity 
ambiguity  removal  for  both  software  and  hardware,  and  cooling.  Clutter 
rejection  capability  should  be  evaluated  on  the  basis  of  a  specified  target 
size  in  a  specified  type  of  terrain.  Verify  that  large  point  targets  do 
not  cause  false  alarms  in  the  automatic  detection  system.  Evaluate  the 
data  displays  in  the  airborne  environment,  and  the  receiver  design  including 
factors  such  as  ECCM  (vulnerability  of  the  system  to  countermeasures  should 
be  evaluated),  dynamic  range,  sensitivity  requirements,  and  false  alarm 
threshholds.  The  T&E  should  include  the  effects  of  such  factors  as  frame 
time  and  dwell  time  and  should  be  conducted  using  ont  only  the  primary 
modes  but  the  backup  modes,  degraded  modes,  and  manual  modes.  Ground 
support  equipment  should  be  evaluated  in  conjunction  with  the  testing 
of  the  airborne  equipment. 

2.  AVOID  TESTING  DELAYS 

When  testing  is  delayed  because  of  the  non-availability  of  critical 
sub-system  components,  off-the-shelf  interim  components  should  be 
used  as  substitutes  until  the  proper  components  are  available. 

As  long  as  the  off-the-shelf  components  can  function  acceptably  within 
a  defined  range  of  interest,  the  rest  of  the  system  can  be  tested,  thereby 
facilitating  the  progress  of  the  test  program.  Clearly,  this  cannot  be  con¬ 
tinued  indefinitely,  but  it  should  serve  to  reduce  the  lingering  influence 
of  long  lead  time  components  that  are  not  available  on  time.  Selected 
tests  may  have  to  be  repeated  when  the  proper  component  is  available. 
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For  example,  it  may  be  possible  to  carry  out  meaningful  system  testing 
without  having  a  full  power  transmitter;  clutter  rejection  tests,  tests  of 
antenna  side  lobe  interaction  with  signal  processor,  and  demonstration  of  th 
range/velocity  ambiguity  removal  algorithms  in  an  actual  clutter  environ¬ 
ment  may  all  be  done  without  a  full  power  transmitter. 

3.  TEMPERATURE  CYCLE  TESTS 

Special  attention  should  be  devoted  to  temperature  cycle  test  results 
since  they  will  help  identify  component  and  structural  interface  pro¬ 
blems  . 

Specifically,  the  temperature  cycling  tests  should  simulate  the 
expected  temperature  environments  with  respect  to  the  numbers  of  and  peak 
excursions  of  the  cycles . 

4.  INTERFACES  WITH  OTHER  SYSTEMS 

Whenever  possible,  the  IOT&E  (as  well  as  the  follow-on  OT&E)  of  an  air¬ 
borne  general  surveillance  radar  should  be  planned  to  include  any  other 
systems  which  must  have  a  technical  interface  with  the  new  system. 

For  example,  radar  equipment  should  be  tested  on  all  the  existing 

aircraft  for  which  it  has  been  programmed,  and  the  interface  with  the  perti- 
3 

nent  C  should  be  tested.  Whenever  more  than  one  system  is  involved, 
specific  attention  should  be  directed  to  the  interfaces  of  the  systems, 
especially  the  display  and  control  subsystems. 


5.  MECHANICAL  COMPONENTS 

Ground  test  all  mechanical  components  of  an  airborne  general  surveillance 
radar  system  prior  to  installation  in  the  aircraft.  Repeat  testing  in 
the  airborne  environment . 


This  includes  items  such  as  rotary  joints,  the  radome  itself,  the 
drive  system,  the  angle  sensor  and  any  other  mechanical  components.  It 
must  be  remembered  that  aerodynamic  loads  as  well  as  gravity  loads  are 
important  in  the  airborne  environment. 
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6.  GROUND  TEST 


Where  available,  it  Is  highly  desirable  to  have  a  hot  bench  on  the 
ground  which  can  be  used  to  supplement  the  airborne  system. 

The  purpose  of  a  hot  bench  is  to  be  able  to  test  items  which  are  being 
run  in  the  air  without  the  necessity  for  an  actual  flight.  Faulty  compo¬ 
nents  may  then  be  identified  and  replaced  prior  to  performance  of  a  test. 

7.  MAINTENANCE  OF  FLEXIBILITY 

Be  sure  that  the  production  design  stage  has  been  reached  before  the 
flexibility  of  the  brassboard  version  is  eliminated. 

This  includes  any  preliminary  attempts  at  cutting  weight  which  may 
unfavorably  influence  the  flexibility  capabilities. 

8.  VALUE  OF  ENVELOPE  EXPANSION 

Contract  requirements  and  incentives  should  not  be  based  on  extreme 
corners  of  the  theoretical  performance  envelope  unless  there  is  a  high 
pay-off  for  such  performance. 

For  example,  considerable  extra  test  effort,  time,  and  money  might  be 
spent  on  getting  completely  satisfactory  clutter  rejection  performance 
over  urban  localities  when  this  environment  is  not  usually  encountered. 

Thus  a  contract  that  blindly  specifies  that  the  clutter  rejection  character¬ 
istics  must  be  fully  compliant  throughout  the  performance  envelope  may 
lead  to  a  non-cost  effective  design. 

At  the  same  time,  the  contract  incentive  fee  criteria  should  not 
constrain  the  developer  from  exploring  the  full  performance  envelope.  For 
example,  inability  to  reject  ground  moving  targets  should  not  preclude 
testing  of  this  capability  merely  because  it  will  reduce  payments  or 
jeopardize  continuation  of  the  program.  In  addition,  do  not  allow  per¬ 
formance  test  criteria  to  be  limited  to  an  oversimplified  evaluation  cri¬ 
terion,  such  as  whether  or  not  the  target  can  be  detected  by  the  radar 
under  ideal"look-up"  conditions. 
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9. 


MAINTENANCE  DATA  PACKACE 


Prior  to  the  decision  to  go  into  full-scale  production  of  an  airborne 
general  surveillance  radar  system,  a  complete  technical/maintenance 
data  package  must  be  prepared  and  tested  to  ensure  that  the  system  can 
be  maintained. 

The  testing  of  this  package  should  be  considered  as  an  essential  part 
of  DT&E  and  also  as  an  essential  part  of  the  IOT&E  of  the  system.  Criteria 
for  successful  demonstration  of  this  package  should  be  established  in  both 
types  of  tests. 

* 

10.  PRODUCTION  DECISION 

Conduct  IOT&E  before  making  a  production  decision.  Anticipate  pro¬ 
duction  problems  it  testing  is  still  being  done  after  initiation  of 
production. 

Because  of  the  need  for  early  delivery  of  an  airborne  general  surveil¬ 
lance  radar  unit,  development  testing  was  done  on  a  continuing  basis  during 
early  production.  Testing  overlapped  approximately  the  first  10  percent 
of  the  sets  which  were  eventually  built .  As  problems  were  identified  most 
of  them  could  be  cured  but  at  the  expense  of  production  efficiency.  Changes 
were  made  during  the  production  run  to  incorporate  fixes  which  were  re¬ 
quired,  but  this  tended  to  interfere  with  a  smooth  running  production  line. 
In  this  case  loss  of  production  efficiency  was  accepted  in  order  to  meet 
time  oriented  milestones. 


IV.  SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 


This  phase  occurs  after  the  DSARC  substantial  production  decision. 
Follow-on  OT&E  will  be  conducted  with  production  hardware.  It  is 
suggested  that  the  checklists  for  the  previous  phases,  especially  the 
last  phase,  be  reviewed  since  many  of  the  items  will  be  applicable  for 
this  phase.  The  primary  checklist  item  for  this  phase  is: 

FOLLOW-ON  OT&E 

The  follow-on  OT&E  plan  should  include  tests  of  any  operational  modes 

not  previously  tested  in  IOT&E. 

All  operational  modes,  including  backup  modes,  should  be  tested  in  the 
follow-on  OT&E  because  the  software  interface  with  the  production  hardware 
system  should  be  thoroughly  evaluated.  Otherwise,  small  easy-to-fix 
problems  could  seriously  impair  overall  system  performance  under  adverse 
conditions. 

The  test  phase  should  be  extended  if  necessary  to  evaluate  system 
adequacy  in  the  face  of  new  operational  modes  that  become  known  late.  The 
follow-on  OT&E  activity  and  the  continuous  operational  testing  that  usually 
follow  system  deployment  must  be  coordinated  to  make  most  effective  use  of 
resources  in  testing  new  operational  modes. 
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